Methods and apparatus for efficiently sharing memory and processing in a multi-processor

ABSTRACT

A shared memory network for communicating between processors using store and load instructions is described. A new processor architecture which may be used with the shared memory network is also described that uses arithmetic/logic instructions that do not specify any source operand addresses or target operand addresses. The source operands and target operands for arithmetic/logic execution units are provided by independent load instruction operations and independent store instruction operations.

RELATED U.S. APPLICATION DATA

The present application claims the benefit of U.S. ProvisionalApplication No. 60/665,668 filed Mar. 28, 2005 and U.S. ProvisionalApplication No. 60/687,719 filed Jun. 6, 2005, both of which areincorporated by reference herein in their entirety.

FIELD OF INVENTION

The present invention relates to unique and improved methods andapparatuses for processor architecture and organizations of processorsand memory modules such that communication between the modules isefficient. More specifically, this invention concerns multiprocessorsystems having a shared memory interconnection network for communicationamong the processors and memory modules and an architecture used by theprocessors that efficiently supports such communication.

BACKGROUND OF INVENTION

One of the problems associated with increasing performance inmultiprocessor parallel processing systems is the efficient accessing ofdata or instructions from memory. Having adequate memory bandwidth forsharing of data between processors is another problem associated withparallel processing systems. These problems are related to theorganization of the processors and memory modules and the processorarchitecture used for communication between a processor and memory andbetween processors. Various approaches to solving these problems havebeen attempted in the past, for example, array processors and sharedmemory processors.

Multiprocessor systems can be classified generally in terms of couplingstrength for communication between processors. Those multiprocessorsystems that communicate using a share memory facility between theprocessors and the shared memory over an interconnection network aregenerally considered tightly coupled. Loosely coupled multiprocessorsystems generally use an input/output (I/O) communication mechanism ineach processor, such as message passing, for communicating between theprocessors over an interconnection network. A wide variety ofinterconnection networks have been utilized in multiprocessing systems.For example, rings, bus connected, crossbar, tree, shuffle, omega, andbutterfly, mesh, hypercube, and ManArray networks, have been used inprior multiprocessor systems. From an application or use perspective,specific networks have been chosen primarily based upon performancecharacteristics and cost to implement tradeoffs.

A network for an application of a multiprocessor system is evaluatedbased on a number of characteristics. Parameters considered include, forexample, a network size of N nodes, where each node has L connectionlinks including input and output paths, a diameter D for the maximumshortest path between any two pair of nodes, and an indication of dacost C in terms of the number of connection paths in the network. A ringnetwork, for example, provides connections between adjacent processorsin a linear organization with L=2, D=N/2, and C=N. In another example, acrossbar switch network provides complete connectivity among the nodeswith L=N, D=1, and C=N². Table 1 illustrates these characteristics for anumber of networks where N is a power of 2.

Network of N nodes Links Diameter N a power of 2 (L) (D) Cost (C) Ring 2N/2 N BxB Torus for N = 2^(K) 4 B = 2^(K/2) 2N For K even & B = 2^(K/2)XD Hypercube for Log₂N Log₂N (X/2)N X = Log₂N XD ManArray hypercube 4 22^(2k−1)((4 + 3^(k−1))−1) for X = 2k and X even Crossbar N 1 N²

FIG. 1A illustrates a prior art 4×4 torus network 100 having sixteenprocessor (P) elements (PEs). Each PE supports four links in the regularnearest neighborhood connection pattern shown. The diameter is four,which is the maximum shortest path between any two nodes, such as, forexample, P00 104 and P22 108. The cost is thirty-two representing thethirty-two connections used to interconnect the PEs.

FIG. 1B illustrates a connectivity matrix 150 for the 4×4 torus network100 of FIG. 1A. Each of the sixteen PEs represents a column and a row ofthe matrix. A “1” in a cell of the connectivity matrix 150 indicatesthat the row PE connects to the column PE. For example, four “1”spopulate P21 row 154, indicating that P21 connects to P11, P20, P22, andP31. The connectivity matrix 150 is populated only with the nearestneighbor connections.

FIG. 2 illustrates a prior art 4×4 ManArray network 200, as illustratedin U.S. Pat. No. 6,167,502. The 4×4 ManArray network 200 has sixteenprocessors such as processor 1, 3 (0110) 204. Each processor isconnected to a local cluster switch, such as local cluster switch 208associated with a 2×2 processor cluster, such as, 2×2 processor cluster212. In the cluster switch are a number of multiplexers which areconnected to the processors to provide the interconnecting network forthe sixteen processors. For example, each of the four processors in the2×2 processor cluster 212 connect to four multiplexers in the associatedlocal cluster switch 208. The 4×4 ManArray network 200 has an indicationof the cost C of 88 and a diameter of 2.

FIG. 3 illustrates a prior art shared memory processor 300 havingprocessor nodes P0-Pp-1 304, memory nodes M0-Mm-1 306, input/output(I/O) nodes I/O0-I/Od-1 308 interconnected by a cross bar switch 310.The cross bar switch provides general data accessing between theprocessors, memory, and I/O. The processors typically interface tomemory over a memory hierarchy which typically locates instruction anddata caches local to the processors. The memories M0-Mm-1 typicallyrepresent higher levels of the memory hierarchy above the local caches.

The prior techniques of interconnecting memory and processors have tocontend with multiple levels of communication mechanisms and complexorganizations of control and networks.

SUMMARY OF THE INVENTION

It is appreciated that improvements to processor architecture, networkdesign, and organizations of processors and memory are desired. Suchimprovements are provided by multiple embodiments of the presentinvention. In one embodiment of the present invention a network of nodesidentified according to a G×H matrix with gε{0, 1, . . . , G−1} andhε(0, 1, . . . , H−1) is provided. The network of nodes has a first setof nodes {A_(0,0), A_(0, 1), . . . , A_(0,H-1), A_(1,0), A_(1, 1), . . ., A_(1, H-1), . . . , A_(G-1,H-1)} where each node A_(g,h) has anoutput. A second set of nodes {R_(0,0), R_(0, 1), . . . , R_(0,H-1),R_(1,0), R_(1, 1), . . . , R_(1,H-1), . . . , R_(G-1,H-1)} where eachnode R_(g,h) has an output and each node R_(g,h) has a first inputconnected to the output of node A_(g,h), a second input connected to theoutput of node A_(g,h+1), and a third input connected to the output ofnode A_(g,h−1), where h+1 wraps to 0 when h+1=H and h−1 wraps to H−1when h−1=−1. The network of nodes also has a third set of nodes{S_(0,0), S_(0, 1), . . . , S_(0,H-1), S_(1,0), S_(1, 1), . . . ,S_(1,H-1), . . . , S_(G-1,H-1)} where each node S_(g,h) has an outputand each node S_(g,h) has a first input connected to the output of nodeR_(g,h), a second input connected to the output of node R_(g+1,h), and athird input connected to the output of node R_(g−1,h), where g+1 wrapsto 0 when g+1=G and g−1 wraps to G−1 when g−1=−1.

Another embodiment of the present invention provides a method ofconstructing a network of nodes N(i)_(g,h) identified by gε{0, 1, . . ., G−1}, hε{0, 1, . . . , H−1}, where N(0) identifies nodes in a firstset of nodes, N(1) identifies nodes in a second set of nodes, and N(2)identifies nodes in a third set of nodes. The method provides steps forconnecting for i=0 an output of each node N(i)_(g,h) to an input of nodeN(i+1)_(g,h) and to an input of node N(i+1)_(g,h+1) and to an input ofnode N(i+1)_(g,h−1) where h+1 wraps to 0 when h+1=H and h−1 wraps to H−1when h−1=−1. A step for connecting for i=1 an output of each nodeN(i)_(g,h) to an input of node N(i+1)_(g,h) and to an input of nodeN(i+1)_(g+1,h) and to an input of node N(i+1)_(g−1,h) where g+1 wraps to0 when g+1=G and g−1 wraps to G−1 when g−1=−1.

A further embodiment of the present invention has a processor nodeoperative to execute arithmetic instructions. The processor has anarithmetic instruction decoder that decodes arithmetic instructionsresponsive to arithmetic instructions having a format containing onlyexecution unit operation information without use of source operand ortarget operand fields. The processor also has an execute unit thatexecutes the decoded arithmetic instruction with source operandsprovided as an input to the arithmetic execution unit, the sourceoperands specified by a load instruction independent of the arithmeticinstruction and the execution unit generates results of the arithmeticexecution that are provided as an output of the arithmetic executionunit, the storage of the results specified by a store instructionindependent of the arithmetic instruction.

These and other features, aspects, techniques and advantages of theinvention will be apparent to those skilled in the art from thefollowing detailed description, taken together with the accompanyingdrawings and claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a prior art 4×4 torus network having sixteenprocessing elements (PEs);

FIG. 1B illustrates a connectivity matrix for the 4×4 torus network ofFIG. 1A;

FIG. 2 illustrates a prior art 4×4 ManArray network from U.S. Pat. No.6,167,502;

FIG. 3 illustrates a prior art shared memory processor;

FIG. 4A illustrates a Wings array memory (WAM) sixteen processor (16)network for store (S) operations in accordance with the presentinvention;

FIG. 4B illustrates the effective store connectivity of the WAM16Snetwork of FIG. 4A in accordance with the present invention;

FIG. 5A illustrates a WAM16 load (L) network for load operations inaccordance with the present invention;

FIG. 5B illustrates the effective load connectivity of the WAM16Lnetwork of FIG. 5A in accordance with the present invention;

FIG. 6A illustrates a connectivity matrix for store operations for theWAM16S network of FIG. 4A in accordance with the present invention;

FIG. 6B illustrates a connectivity matrix for load operations for theWAM16L network of FIG. 5A in accordance with the present invention;

FIG. 6C illustrates a connectivity matrix for communicating betweenprocessors by combining store WAM16S and load WAM16L operations inaccordance with the present invention;

FIG. 7 illustrates an alternative WAM16L network for the purpose ofshowing the symmetric nature of the WAM network in accordance with thepresent invention;

FIG. 8A illustrates a construction of a WAM network node using a four toone multiplexer with a fan out to three locations in accordance with thepresent invention;

FIG. 8B illustrates an alternative construction of a WAM network nodeusing three four to one multiplexers each with a single fan out to aseparate location in accordance with the present invention;

FIG. 9A illustrates a WAM sixty-four processor (64) store (WAM64S)network showing the scalable nature of the Wings array memory network inaccordance with the present invention;

FIG. 9B illustrates a general form of a store path selected from theWAM64S network of FIG. 9A in accordance with the present invention;

FIG. 9C illustrates a store path selected from the WAM64S network ofFIG. 9A in accordance with the present invention;

FIG. 9D illustrates a three dimensional organization of the twenty sevenmemory nodes and processor P_(2,2,2) of FIG. 9C in accordance with thepresent invention;

FIG. 9E illustrates a method of constructing a network in accordancewith the present invention;

FIG. 10A illustrates a generic type of prior art arithmetic instructionformat;

FIG. 10B illustrates a Wings basic arithmetic/logic instruction formatin accordance with the present invention;

FIG. 10C illustrates a Wings basic store instruction format inaccordance with the present invention;

FIG. 10D illustrates a Wings basic load instruction format in accordancewith the present invention;

FIG. 10E illustrates a Wings basic load immediate format in accordancewith the present invention;

FIG. 11A illustrates a Wings processor node for use with the WAMnetworks and using the Wings basic instruction formats in accordancewith an embodiment of the present invention;

FIG. 11B illustrates an example of a WAM processor system in accordancewith the present invention;

FIG. 11C illustrates a WAM16 processor subsystem with a set of processornodes, a WAM16S/WAM16L combined network, a first set of memories, and asecond set of memories in accordance with the present invention;

FIG. 11D illustrates a combined network node that combines a WAM16L nodeand a WAM16S node into a single node and illustrates the function aspectof the WAM nodes in accordance with the present invention;

FIG. 12A illustrates Wings processor node made up of an execution nodeand a memory node in accordance with an embodiment of the presentinvention; and

FIG. 12B illustrates processor node made up of an execution node and amemory node in accordance with an embodiment of the present invention.

DETAILED DESCRIPTION

FIG. 4A illustrates a Wings array memory (WAM) sixteen processor (16)(WAM16S) network 400 for store (S) operations. A processor array 404 ofsixteen processors 405-420 are illustrated as nodes that each caninitiate a store operation to store data in a memory location in theWings array memory (WAM) 424 consisting of sixteen memory blocks425-440. The processor and memory block nodes are organized in lineararrays and identified according to a G×H matrix where, in this example,G equals four representing the number of rows in the matrix and H equalsfour representing the number of columns. A processor P_(g,h), a memoryblock M_(g,h), and internal nodes of the network are labeled in a row gby column h format where gε{0, 1, . . . , G−1} and hε{0, 1, . . . ,H−1}. The processors are not directly connected to each other nor arethe memory blocks directly connected to any of the other memory blocks.

A two stage WAM network 444 interconnects the processors 405-420 andmemory blocks 425-440 for store operations. A first stage of nodes aremultiplexers 445-460 which are labeled in a row g by column h R_(g,h),matrix. A second stage of nodes are multiplexers 465-480 which arelabeled in a row g by column h S_(g,h) matrix. The processors P_(g,h)each have an output, memory blocks M_(g,h) each have an input, andmultiplexers R_(g,h) and S_(g,h) each have three inputs and an output.The processors P_(g,h), the memory bocks M_(g,h), the multiplexersR_(g,h), and the multiplexers S_(g,h) are labeled in the figures as Pgh,Mgh, Rgh, and Sgh, respectively, for ease of notation and reference inthe figures. The first stage of multiplexers 445-460 are partitionedinto groups by rows of the G=4×H=4 matrix. For example, in the first rowg=0 of the processor matrix, the outputs of the processors 405-408 areconnected to the inputs of the multiplexers 445-448. For the next row,g=1, the outputs of the processors 409-412 are connected to the inputsof the multiplexers 449-452. The next row, g=2, the outputs of theprocessors 413-416 are connected to the inputs of the multiplexers453-456. The last row, g=3, processors 417-420 are connected tomultiplexers 457-460.

In each group, the connections are made according to an adjacency ofnodes in a first dimension, for example, P00 405 is connected to R00445, R01 446, and R03 448. P01 406 is connected to R00 445, R01 446, andR02 447. P02 407 is connected to R01 446, R02 447, and R03 448. P03 408is connected to R00 445, R02 447, and R03 448. Each processor in thesecond row group P10-P13 409-412, third row group P20-P23 413-416, andfourth row group P30-P33 417-420, are connected in a similar fashionaccording to their row adjacency to second row multiplexers R10-R13449-452, third row multiplexers R20-R23 453-456, and fourth rowmultiplexers R30-R33 457-460, respectively.

The first stage multiplexers 445-460 are connected to the second stagemultiplexers 465-480 according to an adjacency of nodes in a seconddimension, for example, the output of the multiplexer node R00 445 isconnected to the inputs of the multiplexer nodes S00 465, S10 469, andS30 477. In a similar fashion, R01 446 is connected to S01 466, S11 470,and S31 478. R02 447 is connected to S02 467, S12 471, and S32 479. R03448 is connected to S03 468, S13 472, and S33 480. The multiplexers inthe second row group R10-R13 449-452 are connected to the second stagemultiplexers according to their column adjacency, such that, R10 449 isconnected to S00 465, S10 469, and S20 473, R11 450 is connected to S01466, S11 470, and S21 474, R12 451 is connected to S02 467, S12 471, andS22 475, and R13 452 is connected to S03 468, S13 472, and S23 476. Thethird row group R20-R23 453-456 and the fourth row group R30-R33 457-460are connected in a similar fashion according to their column adjacencyassociated second stage multiplexers from the multiplexers 465-480.

Each output of the second stage multiplexers connects to the input oftheir associated memory block at the same row column position. Forexample, the output of the multiplexer S00 465 connects to the input ofthe memory block M00 425, the output of the multiplexer S01 466 connectsto the input of the memory block M01 426, and so forth. A processorexecuting a store operation can write data to a single memory block orcombinations of up to nine memory blocks from the memory array 424. Forexample, processor P21 can store data to memories in its connected groupof memory blocks M10 429, M20 433, M30 437, M11 430, M21 434, M31 438,M12 431, M22 435, and M32 439.

The adjacency of nodes is represented by a G×H matrix where the nodes ofthe matrix, may be processors, memory blocks, multiplexers, or the like,generally, having nodes N_(g,h) where gε{0, 1, . . . , G−1} and hε{0, 1,. . . , H−1}. A connection network, such as, the WAM16S network 400 ofFIG. 4A, may be generalized as having a first set of nodes, such asprocessor nodes P_(g,h), for example, connects to a second set of nodesR_(g,h) which connects to a third set of nodes S_(g,h). The third set ofnodes S_(g,h) then connects to a fourth set of nodes, such as memoryblock nodes M_(g,h), for example. The store connectivity of the nodescan be viewed as having nodes R_(g,h) connect as follows:

Inputs Connects to the of Node outputs of the Nodes Where R_(g, h)P_(g, h), P_(g, h+1), and h + 1 wraps to 0 when h + 1 = H and P_(g, h−1)h − 1 wraps to H − 1 when h − 1 = −1The nodes S_(g,h) connect as follows:

Inputs Connects to the of Node outputs of the Nodes Where S_(g, h)R_(g, h), R_(g+1,) _(h), and g + 1 wraps to 0 when g + 1 = G andR_(g−1,) _(h) g − 1 wraps to G − 1 when g − 1 = −1The nodes M_(g,h) connect as follows:

Input of Node Connects to the output of the Node M_(g, h) S_(g, h)

For the example WAM16S network 400 of FIG. 4A, the nodes R_(g,h) connectas follows:

Inputs Connects to the of Node outputs of the Nodes Where R_(g, h)P_(g, h), P_(g, h+1), and h + 1wraps to 0 when h + 1 = 4 and P_(g, h−1)h − 1 wraps to 4 − 1 = 3 when h − 1 = −1The nodes S_(g,h) connect as follows:

Inputs Connects to the of Node outputs of the Nodes Where S_(g, h)R_(g, h), R_(g+1,) _(h), and g + 1 wraps to 0 when g + 1 = 4 andR_(g−1,) _(h) g − 1 wraps to 4 − 1 = 3 when g − 1 = −1The nodes M_(g,h) connect as follows:

Input of Node Connects to the output of the Node M_(g, h) S_(g, h)

The store connectivity of the nodes can also be viewed as having nodesP_(g,h) connect as follows:

Output Connects to an of Node input of the Nodes Where P_(g, h)R_(g, h), R_(g, h+1), and h + 1 wraps to 0 when h + 1 = H and R_(g, h−1)h − 1 wraps to H − 1 when h − 1 = −1The nodes R_(g,h) connect as follows:

Output Connects to an of Node input of the Nodes Where R_(g, h)S_(g, h), S_(g+1, h), and g + 1 wraps to 0 when g + 1 = G and S_(g−1, h)g − 1 wraps to G − 1 when g − 1 = −1The nodes S_(g,h) connect as follows:

Output of Node Connects to the input of the Node S_(g, h) M_(g, h)

This store connectivity is more clearly shown in FIG. 4B whichillustrates the effective store connectivity 485 of the WAM16S network400 of FIG. 4A. FIG. 4B is an overhead view of the memory array 424 ofFIG. 4A (octagonal blocks) overlaid upon the processor array 404 of FIG.4A (square blocks). The effective store paths between processors andmemories are obtained through the use of the two stage WAM network 444of FIG. 4A. Such effective store paths are shown as arrow linesconnecting a processor to a memory block. A store path between processorP_(g,h) and memory M_(g,h), such as between P21 414 and M21 434, isshown as a short arrow line beginning from the processor label P_(g,h)and pointing to the memory M_(g,h) block. Each memory block can bereached for storing data from a neighborhood of nine processors.

FIG. 5A illustrates a Wings array memory (WAM) sixteen processor (16)(WAM16L) network 500 for load (L) operations. A processor array 504 ofsixteen processors 505-520 are illustrated as nodes that each caninitiate a load operation to fetch data from a memory location in theWings array memory (WAM) 524 consisting of sixteen memory blocks525-540. The processor and memory block nodes are organized in a lineararray and identified according to a G×H matrix where G equals fourrepresenting the number of rows in the matrix and H equals fourrepresenting the number of columns. A processor P_(g,h) and a memoryblock M_(g,h) are labeled in a row g by column h format where gε{0, 1, .. . , G−1} and hε{0, 1, . . . , H−1}. The processors are not directlyconnected to each other nor are the memory blocks directly connected toany of the other memory blocks.

A two stage WAM network 544 interconnects the processors 505-520 andmemory blocks 525-540 for load operations. A first stage of nodes aremultiplexers 545-560 which are labeled in a row column T_(g,h) matrix. Asecond stage of nodes are multiplexers 565-580 which are labeled in arow column L_(g,h) matrix. The processors P_(g,h) each have an input,memory blocks M_(g,h) each have an output, and multiplexers T_(g,h) andL_(g,h) each have three inputs and an output. The processors P_(g,h),the memory bocks M_(g,h), the multiplexers T_(g,h), and the multiplexersL_(g,h) are labeled in the figures as Pgh, Mgh, Tgh, and Lgh,respectively, for ease of notation and reference in the figures. Thefirst stage of multiplexers 545-560 are partitioned into groups by rowsof the G=4×H=4 matrix. For example, in the first row g=0 of the memorymatrix, memories 525-528 are connected to multiplexers 545-548. For thenext row, g=1, memories 529-532 are connected to multiplexers 549-552.The next row, g=2, memories 533-536 are connected to multiplexers553-556. The last row, g=3, memories 537-540 are connected tomultiplexers 557-560.

In each group, the connections are made according to an adjacency ofnodes in a first dimension, for example, M00 525 is connected to T00545, T01 546, and T03 548. M01 526 is connected to T00 545, T01 546, andT02 547. M02 527 is connected to T01 546, T02 547, and T03 548. M03 528is connected to T00 545, T02 547, and T03 548. Each memory block in thesecond row group M10-M13 529-532, third row group M20-M23 533-536, andfourth row group M30-M33 537-540, are connected in a similar fashionaccording to their row adjacency to second row multiplexers T10-T13549-552, third row multiplexers T20-T23 553-556, and fourth rowmultiplexers T30-T33 557-560, respectively.

The first stage multiplexers 545-560 are connected to the second stagemultiplexers 565-580 according to an adjacency of nodes in a seconddimension, for example, T00 545 is connected to L00 565, L10 569, andL30 577. T01 546 is connected to L01 566, L11 570, and L31 578. T02 547is connected to L02 567, L12 571, and L32 579. T03 548 is connected toL03 568, L13 572, and L33 580. The multiplexers in the second row groupT10-T13 549-552 are connected to the second stage multiplexers accordingto their column adjacency, such that, T10 549 is connected to L00 565,L10 569, and L20 573, T11 550 is connected to L01 566, L11 570, and L21574, T12 551 is connected to L02 567, L12 571, and L22 575, and T13 552is connected to L03 568, L13 572, and L23 576. The third row groupT20-T23 553-556 and the fourth row group T30-T33 557-560 are connectedin a similar fashion according to their column adjacency associatedsecond stage multiplexers.

Each output of the second stage multiplexers connects to the load inputof their associated processors at the same row column position. Forexample, the output of the multiplexer L00 565 connects to the input ofprocessor P00 505, the output of the multiplexer L01 566 connects to theinput of processor P01 506, and so forth. A processor executing a loadoperation can select a memory block from a group of nine memory blocksto fetch data from the selected memory block. For example, processor P21514 can load data from memories in its connected group of memory blocksM10 529, M20 533, M30 537, M11 530, M21 534, M31 538, M12 531, M22 535,and M32 539. Load addresses may follow connection paths in a networkconfiguration such as the WAM16S network 400 of FIG. 4A, for example toprovide memory addresses to selected memories as part of a loadoperation. Alternative methods to handle address paths is discussed inmore detail below.

The adjacency of nodes is represented by a G×H matrix where the nodes ofthe matrix, may be processors, memory blocks, multiplexers, or the like,generally, having nodes N_(g,h) where gε{0, 1, . . . , G−1} and hε{0, 1,. . . , H−1}. A connection network, such as, the WAM16L network 500 ofFIG. 5A, may be generalized as having a first set of nodes, such asmemory nodes M_(g,h), for example, connects to a second set of nodesT_(g,h) which connects to a third set of nodes L_(g,h). The third set ofnodes L_(g,h) then connects to a fourth set of nodes, such as processornodes P_(g,h), for example. The load connectivity of the nodes can beviewed as having nodes T_(g,h) connect as follows:

Inputs Connects to the of Node outputs of the Nodes Where T_(g, h)M_(g, h), M_(g, h+1), and h + 1 wraps to 0 when h + 1 = H and M_(g, h−1)h − 1 wraps to H − 1 when h − 1 = −1The nodes L_(g,h) connect as follows:

Inputs Connects to the of Node outputs of the Nodes Where L_(g, h)T_(g, h), T_(g+1, h), and g + 1 wraps to 0 when g + 1 = G and T_(g −1,)_(h) g − 1 wraps to G − 1 when g − 1 = −1The nodes P_(g,h) connect as follows:

Input of Node Connects to the output of the Node P_(g, h) L_(g, h)

For the example WAM16L network 500 of FIG. 5A, the nodes T_(g,h) connectas follows:

Inputs Connects to the of Node outputs of the Nodes Where T_(g, h)M_(g, h), M_(g, h+1), and h + 1 wraps to 0 when h + 1 = 4 and M_(g, h−1)h − 1 wraps to 4 − 1 = 3 when h − 1 = −1The nodes L_(g,h) connect as follows:

Inputs Connects to the of Node outputs of the Nodes Where L_(g, h)T_(g, h), T_(g+1,) _(h), and g + 1 wraps to 0 when g + 1 = 4 andT_(g−1,) _(h) g − 1 wraps to 4 − 1 = 3 when g − 1 = −1The nodes P_(g,h) connect as follows:

Input of Node Connects to the output of the Node P_(g, h) L_(g, h)

This load connectivity is more clearly shown in FIG. 5B whichillustrates the effective load connectivity 585 of the WAM16S network500 of FIG. 5A. FIG. 5B is an overhead view of the processor array 504of FIG. 5A (square blocks) overlaid upon the memory array 524 of FIG. 5A(octagonal blocks). The effective load paths between memories andprocessors are obtained through the use of the two stage WAM network 544of FIG. 5A. Such effective load paths are shown as arrow linesconnecting a memory block to a processor. A load path between memoryM_(g,h) and processor P_(g,h), such as between M21 534 and P21 514, isshown as a short arrow line beginning from the memory M_(g,h) block andpointing to the processor P_(g,h). Each processor can be reached byloading data from a neighborhood of nine memory blocks.

FIG. 6A illustrates a store connectivity matrix 600 for store operationsfor the WAM16S network 400 of FIG. 4A. The processors are organized inthe same linear order as the processor array 404 shown in the WAM16Snetwork 400. The memories are organized in the same linear order as theWings array memory (WAM) 424 shown in the WAM16S network 400. Inaddition to the processor and memory labels used in the WAM16S network400, the processors and memories have a Gray encoded label underneaththe P_(g,h), and M_(g,h) labels. A 1 in a cell of the store connectivitymatrix 600 indicates that a processor on the same row as the cell has astore connection to a memory block on the same column as the cell. Forexample, the connectivity of the processors in processor group 602having processors P10, P11, P12, and P13 connecting to memory blocks inthe three memory block groups 604, 606, and 608 is indicated by “1s” asconnection points in circled connection sub-matrices 610, 612, and 614.

FIG. 6B illustrates a load connectivity matrix 630 for load operationsfor the WAM16L network 500 of FIG. 5A. The processors are organized inthe same order as the processor array 504 in the WAM16L network 500. Thememories are organized in the same linear order as the Wings arraymemory (WAM) 524 shown in the WAM16L network 500. In addition to theprocessor and memory labels used in the WAM16L network 500, theprocessors and memories have a Gray encoded label underneath theP_(g,h), and M_(g,h), labels. A 1 in a cell indicates that a memoryblock on the same row as the cell has a load connection to a processoron the same column as the cell.

FIG. 6C illustrates a connectivity matrix 670 for communicating betweenprocessors by combining store operations on the WAM16S network 400 andload operations on the WAM16L network 500. The connectivity matrix 670is obtained by multiplying the store connectivity matrix 600 with theload connectivity matrix 630. Such multiplication produces thecompletely connected matrix 670 shown in FIG. 6C. The advantageindicated by the completely connected matrix 670 is that completeconnectivity is achieved with less connection cost than a cross barswitch. It is also possible to pipeline stores and loads such that aneffective shortened cycle communication throughput may be obtained whilestill achieving complete connectivity. For example, with store and loadexecution times of a single cycle, an effective single cyclecommunication throughput may be obtained by overlapping store and loadoperations using software pipelining methods.

FIG. 7 illustrates an alternative WAM16L network 700 for the purpose ofshowing the symmetric nature of the WAM network. Both the WAM16L network500 and the WAM16L network 700 have the same load connectivity matrixand can be used interchangeably.

The adjacency of nodes is represented by a G×H matrix where the nodes ofthe matrix, may be processors, memory blocks, multiplexers, or the likehaving nodes N_(g,h) where gε{0, 1, . . . , G−1} and hε{0, 1, . . . ,H−1}. A connection network, such as, the alternative WAM16L network 700of FIG. 7, may be generalized as having a first set of nodes, such asmemory nodes M_(g,h), for example, connects to a second set of nodesT_(g,h) which connects to a third set of nodes L_(g,h). The third set ofnodes L_(g,h) then connects to a fourth set of nodes, such as processornodes P_(g,h), for example. The load connectivity of the nodes can beviewed as having nodes T_(g,h) connect as follows:

Inputs Connects to the of Node outputs of the Nodes Where T_(g, h)M_(g, h), M_(g+1,) _(h), and g + 1 wraps to 0 when g + 1 = G andM_(g−1,) _(h) g − 1 wraps to G − 1 when g − 1 = −1The nodes L_(g,h) connect as follows:

Inputs Connects to the of Node outputs of the Nodes Where L_(g, h)T_(g, h), T_(g, h+1), and h + 1 wraps to 0 when h + 1 = H and T_(g, h−1)h − 1 wraps to H − 1 when h − 1 = −1The nodes P_(g,h) connect as follows:

Input of Node Connects to the output of the Node P_(g, h) L_(g, h)

FIG. 8A illustrates a WAM network node 800 constructed using a three toone multiplexer 802 with a fan out to three locations 804-806. Themultiplexer has three inputs 809-811 as selected by mpx_(gh)(0:1)control signals 812. The states of the control signals 812 are shown incolumnar format 814 inside the multiplexer 802. When the control signals812 are in a specific state, the input associated with that state istransferred to the multiplexer output that fans out to three places804-806. For example, multiplexer control signals 812 set at “10” causethe value on input 810 to be sent to the three fan out locations804-806. The WAM network node 800 would be suitable for using as nodesin the WAM16S Rxx nodes 445-460 of FIG. 4A, Sxx nodes 465-480 of FIG.4A, WAM16L Txx nodes 545-560 of FIG. 5A, Lxx nodes 565-580 of FIG. 5A,alternative WAM16L Txx nodes 745-760 of FIG. 7, and Lxx nodes 765-780 ofFIG. 7.

FIG. 8B illustrates an alternative WAM network node 850 constructedusing three three to one multiplexers 852-854 each with a single fan out856-858 to a separate location. The external inputs 859-861 to thealternative WAM network node 850 have the same source as the inputsignals 809-811 of the WAM network node 800 of FIG. 8A. Each output856-858 of the alternative WAM network node 850 is separately sourced byits associated multiplexer 852-854, respectively. Since there are three3 to 1 multiplexers 852-854 in the alternative WAM network node 850,there are three sets of control signals with two lines each comprisingmp×gh(0:5) 864 required to appropriately control the three multiplexers852-854.

FIG. 9A illustrates a WAM sixty-four processor (64) store (WAM64S)network 900 showing the scalable nature of the Wings array memorynetwork. Each group of 16 processors 902, 904, 906, and 908 areconnected to a WAM16S network 910, 912, 914, and 916, respectively. TheWAM16S networks 910, 912, 914, and 916 are of the same type as theWAM16S network 400. Note that the processor notation, the multiplexernode notation, and the memory notation are based on G×H×K 3 dimensional(3D) cube organization, where G represents the number of rows on aplane, H represents the number of columns on the plane, and K representsthe number of planes in the 3D cube organization. A processor P_(g,h,k),a memory M_(g,h,k), a node R_(g,h,k), a node S_(g,h,k), and a nodeV_(g,h,k) are labeled in a row g by column h by plane k format wheregε{0, 1, . . . , G−1}, hε{0, 1, . . . , H−1}, and kε{0, 1, . . . , K−1}.The WAM 64S network has G=4, H=4, and K=4. The processors P_(g,h,k), thememory bocks M_(g,h,k), the multiplexers R_(g,h,k), the multiplexersS_(g,h,k), and the multiplexers V_(g,h,k), are labeled in the figures asPghk, Mghk, Rghk, Sghk, and Vghk, respectively, for ease of notation andreference in the figures. The WAM64S network has three stages, twostages for the four WAM16S networks 910, 912, 914, and 916 and one stage918 for the K planes that connects to the 64 memory blocks 920, 922,924, and 926. A WAM64L network would be symmetric to the WAM64S network900 in the same manner that the WAM16L network 700 is symmetric to theWAM16S network 400.

FIG. 9B illustrates a general form of a store path 930 selected from theWAM64S network 900. The store path begins at P_(g,h,k) 932 connecting toa first stage 933 of a WAM16S network to three R nodes 934-936. Thethree R nodes 934-936 connect to a second stage 937 of the WAM16Snetwork to nine S nodes 938-946. The nine S nodes 938-946 connectthrough a WAM network stage 947 to twenty seven V nodes 948 that eachconnect directly to a corresponding memory block in the twenty sevenmemory blocks 949.

FIG. 9C illustrates a store path 950 selected from the WAM64S network900. The store path 950 begins at P_(2,2,2) 952. This store path 950 isformed by substituting g=2, h=2, and k=2 in the subscripted notation ofthe general form of a store path 930 in FIG. 9B. The node numbers wrapwithin the range 0-3 for rows g, columns h, and planes k. An examplememory node is memory node M_(3,2,1) 954.

FIG. 9D illustrates a three dimensional organization 960 of the twentyseven memory nodes and processor P_(2,2,2) 952 of FIG. 9C. The storepath begins at P_(2,2,2) 952 and connects to the twenty seven memorynodes, such as memory node M_(3,2,1) 954.

FIG. 9E illustrates a method 970 of constructing a network in accordancewith the present invention. The method starts in step 971 where anetwork of nodes is identified by gε{0, 1, . . . , G−1}, hε{0, 1, . . ., H−1}, kε{0, 1, . . . , K−1}, zε{0, 1, . . . , Z−1} and iε{0, 1, . . ., D} where D is the number of dimensions. In step 972, variables i, g,h, k, . . . , z are initialized to zero.

For i=0 step 974, the first stage of the network is constructedconnecting node N(i)_(g, h, k, . . . , z) to nodeN(i+1)_(g, h, k, . . . , z) and to node N(i+1)_(g,h+1,k, . . . , z) andto N(i+1)_(g, h−1, k, . . . , z) where h+1 wraps to 0 when h+1=H and h−1wraps to H−1 when h−1=−1. In step 978, the variable h is incrementedby 1. In step 979 it is determined whether h=H. If h does not equal H,then the method returns to step 974. If h does equal H, then the methodproceeds to step 980.

In step 980, the variable h is set to zero and the variable g isincremented by 1. In step 981, it is determined whether g=G. If g doesnot equal G, then the method returns to step 974. If g does equal G,then the method proceeds to step 982.

In step 982, the variable g is set to zero and the variable k isincremented by 1. The method 970 continues in like manner for thedimensions up to the test for the last dimension in step 983. In step983, it is determined whether z=Z. If z does not equal Z, then themethod returns to step 974. If z does equal Z, then the method proceedsto step 984.

In step 984, the variable z is set to zero and the variable i isincremented by 1. In step 985, it is determined whether i=D. If i doesnot equal D, then the method proceeds to step 975 with i=1. If i doesequal D, then the method stops at step 986 having constructed thenetwork.

For i=1 step 975, the second stage of the network is constructedconnecting node N(i)_(g, h, k . . . , z) to nodeN(i+1)_(g, h, k, . . . , z) and to node N(i+1)_(g+1, h, k, . . . , z)and to N(i+1)_(g−1, h, k, . . . , z) where g+1 wraps to 0 when g+1=G andg−1 wraps to G−1 when g−1=−1. In step 978, the variable h is incrementedby 1. From step 975, the method proceeds to step 978 and the process isrepeated from step 978 through to the step 984. In step 984, thevariable z is set to zero and the variable i is incremented by 1. Themethod continues constructing stages of the network until the point isreached where i=D−1. In step 985 at this point, the process proceeds tostep 976 to construct the last stage of the network. Once the last stageof the network has been constructed, the method returns to step 984 andincrements the variable i by 1, such that i=D. In step 985, it isdetermined that i=D and the method proceeds to step 986 havingconstructed the network. It is noted that steps 988 are adjusted for thenumber of dimensions D of the network to be constructed. For example, ifD=2, as would be the case for the WAM16S network 400 of FIG. 4A, thenonly variables g and h would be required and steps 982 through steps 983would not be required. Also, step 984 would be adjusted to g=0, i=i+1.

The WAM16S network 400 of FIG. 4A may be constructed by use of themethod 970 where the dimensions (D) is 2. The method 970 would for D=2follow the steps illustrated in FIG. 9E including step 974 and step 975.Step 974 for i=0 and steps 988 adjusted for D=2 are used to constructthe first stage of the WAM16S network 400 between the processors P00 405through P33 420 and the multiplexers R00 445 through R33 460. Step 975for i=1 and steps 988 adjusted for D=2 are used to construct the secondstage of the WAM16S network 400 between the multiplexers R00 445 throughR33 460 and the multiplexers S00 465 through S33 480.

FIG. 10A illustrates a generic type of arithmetic instruction format1000. The arithmetic instruction 1000 is made up of a 6-bit opcode 1002,a 5-bit Rt register target field 1004, a 5-bit Rx register source field1006, a 5-bit Ry register source field 1008, and an 11-bit instructionspecific field 1010. This format is typical for a processor having acentral register file from which arithmetic operands are sourced andarithmetic results are targeted. A thirty two entry register file of,for example, 32-bits, organized as a 32×32-bit multi-port register file,is a typical processor register file requiring 5-bit addressing for eachport for direct access of operands. In a memory to memory processorwhich accesses operands from a memory, the specification of the sourceand target addresses in the arithmetic instruction typicallyaccommodates a wider addressing range. The wider addressing range isobtained either directly through wider operand address fields in aninstruction or through indirect forms of addressing using externaladdressing registers set up ahead of time.

In most processors, a fixed instruction format size is used, such as, 8,16, 24, 32 and 64 bits or a combination of such instruction formats.FIG. 10A shows one such 32-bit instruction format 1000. The spaceallocated in the 32-bit instruction format 1000 for three operandaddress fields 1004, 1006, and 1008 is necessarily limited, since theother instruction bits, such as 1002 and 1010, are required to specifyoperations necessary in order to execute the instruction as specified bythe processor's architecture. In order to break this limitation andprovide greater flexibility in specifying operand addresses, forexample, with greater range and flexible accessing methods, a newprocessor architecture, referenced as the Wings architecture, splits atypical instruction format into three separate new types of instructionformats each more optimally organized for their intended purposes. Afirst instruction format, an arithmetic/logic instruction format 1020,is shown in FIG. 10B to be used to specify arithmetic, logical, shift,bit manipulation, and the like operations. A second instruction format,a store instruction format 1040, is shown in FIG. 10C to be used tospecify operations to store results of arithmetic operations to memory.A third instruction format, a load instruction format 1060, is shown inFIG. 10D to be used to specify the accessing of operand data from memoryfor delivery to execution units. These and other variations arediscussed further below.

For example, FIG. 10B illustrates a Wings basic arithmetic/logic (AL)instruction format 1020 having 12-bits to define the operation. The ALformat 1020 has no memory source or target operand address fields. A6-bit opcode field 1022, a 3-bit data type (Dtype) 1024, and threearithmetic/logic instruction specific bits 1026 are all that is requiredto specify an arithmetic operation in the 12-bit AL instruction format1020. The Wings processor architecture specifies that whatever data isat the inputs to an AL unit at the start of an execution cycle that isthe data received in the AL unit and operated on by the AL unit. TheWings processor architecture also specifies that the results ofexecution are available at the output of the AL unit at the end of theexecution cycle or cycles. An AL instruction does not specify a targetstorage address in a central register file or a memory unit where theresults may be stored. In order to provide operands to an AL unit andstore results from an AL unit, an AL instruction must be paired with aload and a store instruction or other such instruction or instructionsto provide source operands and to take result operands for furtherprocessing or storage.

For example, FIG. 10C illustrates a Wings basic store instruction format1040 having 19-bits to define the operation. The store instructionformat 1040 uses a 3-bit store opcode 1042, two store instructionspecific bits 1044, a 4-bit direction/memory bank (MemBank) selectionfield 1046, and a 10-bit memory address 1048 in the 19-bit instructionformat. As specified by the opcode 1042 or in combination with the storeinstruction specific bits 1044, the store instruction causes a result tobe taken from a specified execution unit and store the result to thetarget memory address. The target memory address is determined from thecombination of the 4-bit direction/MemBank selection field 1046 and the10-bit memory address 1048. Direct, indirect, and other addressing formsmay be specified using separate addressing registers if required.

FIG. 10D illustrates a Wings basic load instruction format 1060 having19-bits to define the operation. The load instruction format uses a3-bit load opcode 1062, two load instruction specific bits 1064, a 4-bitdirection/memory bank (MemBank) selection field 1066, and a 10-bitmemory address 1068 in the 19-bit instruction format. As specified bythe opcode 1062 or in combination with the load instruction specificbits 1064, the load instruction fetches at least one source operand froma specified memory address for delivery to an execution unit. The memoryaddress is determined from the combination of the 4-bitdirection/MemBank selection field 1066 and the 10-bit memory address1068. Direct, indirect, and other addressing forms may be specifiedusing separate addressing registers if required. FIG. 10E illustrates aWings basic load immediate format 1080 having 19-bits to define theoperation. The load immediate format uses a 3-bit load immediate opcode1082 and a 16-bit immediate field 1088 in the 19-bit instruction format.The 3-bit load immediate opcode 1082, for example, may specify theexecution unit that is to use the immediate data.

It is anticipated the depending upon the application the processorarchitecture may expand or contract the illustrated instruction formats.For example, 8-bit arithmetic and 16-bit load and store instructionformats, and 16-bit arithmetic and 24-bit load and store instructionformats can be envisioned, as well as other variations, such as, 14-bitarithmetic and 25-bit load and store instruction formats. Theinstruction format is determined primarily from the number of and typeof operations to be specified for each class of instruction.

A secondary consideration may be how the instructions are packed forstorage as programs in external memory. For example, with use of baseaddress registers local in the PEs, a dual load instruction may bespecified that selects two source operands from blocks of memory bygenerating two addresses. The dual load instruction would be used inplace of two single load instructions. With a dual load instructionformat of 27-bits, a store instruction of 23-bits, and an arithmeticinstruction of 14-bits, a packed instruction storage space of 64-bitswould be required. The packed instruction storage space could beunpacked locally to the processor when loading instruction memories, forexample, as may be specified in direct memory access (DMA) typeoperations. Instruction memories, such as the execution unit instructionmemories of a Wings processor may be used. See U.S. Provisionalapplication Ser. No. 10/648,154 entitled “Methods and Apparatus ForMeta-Architecture Defined Programmable Instruction Fetch FunctionsSupporting Assembled Variable Length Instruction Processors”, which isincorporated by reference in its entirety.

FIG. 11A illustrates a Wings processor node 1100 for use with the WAMnetworks, such as the WAM16S network 400, WAM16L network 500 and 700,and WAM64S network 900. The Wings processor node 1100 uses the Wingsbasic instruction formats, 1020, 1040, 1060, and 1080. The Wingsprocessor node 1100 consists of a processor P_(g,h) 1104 with inputconnections for instruction memory addresses WinF-IM0 address andcontrols 1105, WinF-IM1 address and controls 1106, and WinF-IM2 addressand controls 1107. The processor P_(g,h) 1104 has output connections forWAM network connections 1109-1114 which are described in more detailbelow.

As noted above, the 12-bit arithmetic and 19-bit load and storeinstruction formats are one set of example formats that can be specifiedfor the processor nodes. Depending upon the application, the number andtype of unique instructions may require different instruction formats inorder to meet the requirements. It was also noted that it is desirableto optimize the instruction format to the needs of the instruction type,such as arithmetic/logic instructions, load and store instructions forexample. Since the instruction formats may take various numbers of bits,an architecture supporting a wide variety of formats is required. TheWings architecture, as described in US Patent Application Publication US2004/0039896, is an architecture that would allow different instructionsizes for each instruction type supported by a separate instructionmemory unit. The Wings architecture supplies instruction addresses tolocal instruction memories in each processor, such as load instructionmemory IM0 1116, arithmetic instruction memory IM1 1117, and storeinstruction memory IM2 1118 to select an instruction from each memory.The selected instruction is supplied on individual instruction buses toseparate decode units 1120-1122 and then executed in each separateexecution unit 1124-1126, respectively.

The load execute unit 1124 generates a data fetch address or loadaddress for each load instruction supplied by the load instructionmemory IM0 1116. For example, if two load instructions were suppliedthen two load addresses and network opcodes would be generated, such asload address 1 & load network 1 opcode 1109 and load address 2 & loadnetwork 2 opcode 1110. These fetch addresses and network opcodes are setthrough the network to each multiplexer node that is under control ofthe processor. In the WAM16L network 700, each processor node P_(g,h),for example, controls the network node associated with the direct pathto memory block M_(g,h). For example in FIG. 7, processor P03 708controls nodes L03 768 and T03 748, processor P21 714 controls nodes L21774 and T21 754. In a single instruction multiple data (SIMD) mode ofoperation, each direction associated with a load and store instructionfrom all the nodes involved in the operation provide the same directioncommand code. For example, a load from the east would be specified in abit field of a load instruction and that bit field portion of the loadinstruction would be the same for all load instructions in allprocessors involved in the operation. It is appreciated that differentexecution specific instruction operations such as different directionsof load or store operations may be specified among a group of executingnodes where the communication operations do not conflict. As anotherexample, in a specified group of processor nodes the non-communicationoriented bit field portions of the load instructions may be differentfor each processor node such that data from different memory addressesmay be fetched. When data is returned through the WAM network, it isloaded directly to the arithmetic unit of each processor that is doing aload operation, for example, receiving load data on load operand 1WAMXL1 1111 and load operand 2 WAMXL2 1112.

To associate an arithmetic operation with a load instruction, thelatency of the fetch through the WAM network must be accounted for. Forexample, with a single cycle allocated to address a memory block andobtain the data at the memory block output and a single cycle allocatedto transfer the fetched data across the network to a processor node, twocycles may be used for a data load operation.

Store operations follow a similar path with a store operand data at aspecified memory address is sent through the store WAMXS network to thememory based on the direction command in the store instruction. Thestore operand WAMXS 1113 and store address & store network opcode 1114are sent through the network to the desired memory block for storage.

FIG. 11B illustrates an example of a WAM processor system 1130. G×Hprocessors P_(0,0) 1132, P_(0,1) 1133, . . . , P_(G-1,H-1) 1134 areconnected to a Wings intelligence fetcher (WinF) 1136 through threeinstruction memory address lines 1137-1139. For example, instructionmemory address and control lines 1137-1139. The memory address andcontrol lines are similar to the WinF IM0 address and controls 1105,WinF IM1 address and controls 1106, and WinF IM2 address and controls1107, respectively, as shown in the processor 1100 of FIG. 11A. TheWings intelligent fetcher 1136 fetches its instructions from the Wingsfetch instruction memory (WIM) 1140. The multiple processors connect todata memory through WAM networks, such as two WAMXL load networks,WAMXLA 1142 and WAMXLB 1143, and a WAMXS store network WAMXS1 1144. Withtwo WAM load networks, either multi-port memories or two memory blocksper associated processor node may be used, for example. In FIG. 11B theWAM processor system 1130 uses two memory blocks per associatedprocessor node. For example, there are two memory blocks, MA_(0,0) 1146and MB_(0,0) 1147 associated with processor node P_(0,0) 1132.

FIG. 11C illustrates a WAM16 processor subsystem 1150 with a set ofprocessor nodes 1152, a WAM16S/WAM16L combined network 1153, a first setof memories 1154, and a second set of memories 1155 in accordance withthe present invention. The WAM16S/WAM16L combined network 1153 is madeup of a WAM16S network, such as WAM16S network 400 of FIG. 4A, and aWAM16L network, such as WAM16L network 500. The WAM16S/WAM16L combinednetwork 1153 is used for connecting processor nodes 1152 to the firstset of memories 1154. The second set of memories 1155 connects locallyto the processor nodes 1152. With this organization simultaneous dualmemory loads to the processor nodes 1152 can be supported. Fourprocessor nodes 1156-1159 are illustrated in FIG. 11C that are part of alarger sixteen processor node network, such as illustrated in FIGS. 4Aand 5A, for example. For store operations processor nodes 1156-1159 senddata to the Rxx nodes 1160-1163. For example, processor node P01 1157sends data to R00 1160, R01 1161, and R02 1162. The Rxx nodes 1160-1163connect to Sxx nodes 1164-1167 and other nodes in the WAM16S/WAM16Lcombined network 1153. The Sxx nodes 1164-1167 connect to memories1168-1171, respectively. Though a single block of memory is shown foreach of the memories 1168-1171, it is appreciated that the memories1168-1171 may be partitioned into multiple memory blocks each accessibleby use of addressing ranges. The desired memory block may be specifiedthrough the memory address that is associated with the data being sentthrough the network for storage in memory.

For network load operations, a processor node initiates a network loadoperation by sending a data fetch address and network opcode through thenetwork to the desired memory. The addressed memory fetches data at thespecified address and send the data through the WAM16S/WAM16L combinednetwork 1153 back to the processor node that initiated the network loadoperation, such as one of the processor nodes 1156-1159. The memories1168-1171 are connected to Txx nodes 1172-1175. For example, memory M001168 sends data to T00 1172, T01 1173, and T03 1175. The Txx nodes1172-1175 connect to Lxx nodes 1176-1179 and other nodes in theWAM16S/WAM16L combined network 1153. The Lxx nodes 1176-1179 connect tothe processor nodes 1156-1159, respectively.

For local load operations, a processor node initiates a local loadoperation by sending a data fetch address directly to the local memoryassociated with the processor node. The local memory accesses the dataand provides it locally to the requesting processor node. For example,processor nodes 1156-1159 may load data from local memories 1180-1183,respectively.

Depending upon the application and processor cycle time, it is possibleto store through a WAMXS network into memory in a single cycle and toload data from a memory through a WAMXL network into a processor also ina single cycle. Such performance may be appropriate for low powerapplications, for example. For this type of situation, a softwarepipeline of storing and loading may be easily obtained providing asingle cycle throughput for communicating data between processor nodesfor any node in the system.

FIG. 11D illustrates a combined network node 1185 that combines a WAM16Snode 1186 and a WAM16L 1187 node into a single node 1188. The singlenode 1188 illustrates the functional aspect of the WAM nodes. The WAM16Snode 1186 and WAM16L node 1187 operate under control signal inputsprovided by decoder 1189 and 1190, respectively. The outputs of thedecoders 1189 and 1190 are represented by the binary state lists 1191and 1192, respectively. The decoders 1189 and 1190 receive controlsignals SNOp 1193 and LNOp 1194, respectively. For simple directionalpath control for the data through the networks, the WAM16S node 1186 andWAM16L 1187 node may be multiplexers selecting the path according to thebinary state indicated in the node diagram. In an alternativeembodiment, the control signals SNOp 1193 and LNOp 1194 are useddirectly without need for a decoder. The controls signals SNOp 1193 andLNOp 1194 connect directly to binary multiplexer control inputs that areused for controlling the multiplexers. In another alternativeembodiment, the decoders 1189 and 1190 in select modes of operation passthe control signals through the decoders and providing no additionaldecoding function. For additional functions of the nodes 1186 and 1187,the nodes 1186 and 1187 may provide different operations on data cominginto the nodes, as may be required by an application. These additionalfunctions may be specified by a more complex decoder implementation ofdecoders 1189 and 1190 and an expansion of the control signals SNOp 1193and LNOp 1194. For example, operations on individual data such as shiftoperations may be specified and more complex operations on multipleinput paths, such as compare and addition operations and the like mayalso be specified.

FIG. 12A illustrates Wings processor node 1200 made up of an executionnode 1202 and a memory node 1204 in accordance with an embodiment of thepresent invention. The split organization of the processor node 1200allows the execution node 1202 to be placed at the data input and outputconnections of a WAM store network, such as the WAM16S network 400 ofFIG. 4A and a WAM load network, such as the WAM16L network 500 of FIG.5A. The split organization of the processor node 1200 also allows thememory node 1204 to be placed at the data input and output connectionsof a WAM store network and a WAM load network. A WAM store networkcombined with a WAM load network is represented by network 1206.

The execution node 1202 receives arithmetic/logic instructions over anIM1 instruction bus 1212 connected to an arithmetic decode and executionunit 1214. The arithmetic/logic (AL) instructions each have a formatsuch as the AL instruction format 1020 of FIG. 10B. The received ALinstruction is decoded and executed using source operands XL1DataOut1215 and XL2DataOut 1216 supplied from the network 1206. The arithmeticdecode and execute unit 1214 generates a result XSDataIn 1217 that issent to the network 1206. The AL instruction itself contains no sourceor target operand information.

The memory node 1204 receives store instructions over an IM2 instructionbus 1222 connected to a store decode and execute unit 1224. The storeinstructions each have a format such as the store instruction format1040 of FIG. 10C. The received store instruction is decoded and executedgenerating address lines 1225 that are supplied to memory 1226 andcontrols (XScntls) 1228 supplied to the network 1206. XSDataIn 1217follows the data path of a WAM store network that is part of the network1206 and outputs a XSDataOut 1218. The XSDataOut 1218 is connected tothe memory 1226 and written to memory 1226 when the store instruction isexecuted. The Xscntls 1228 provide multiplexer control signals to thestore portion of the network 1206, such as the WAM16S network 400 ofFIG. 4A, such as multiplexer node 1186 of FIG. 11D.

The memory node 1204 further receives load instructions over an IM0instruction bus 1232 connected to a load decode and execute unit 1234.The load instructions each have a format such as the load instructionformat 1060 of FIG. 10D. The received load instruction is decoded andexecuted generating load address lines to be output to the memory 1226.For dual load instructions, for example, address lines 1235 and 1236 aregenerated. Associated with the generated address lines 1235 and 1236 arecorresponding control lines XL1 cntls 1237 and XL2 cntls 1238,respectively. The XL1 cntls 1237 and XL2 cntls 1238 provide multiplexercontrol signals to the load portion of the network 1206, such as havingtwo WAM16L networks 500 of FIG. 5A and using a multiplexer node, suchas, multiplexer node 1187 of FIG. 11D for each node of the loadnetworks. The two load address lines 1235 and 1236 cause two dataoperands to be read from memory 1226 and output on XL1DataIn 1240 andXL2DataIn 1241 that are connected to the network 1206. The XL1DataIn1240 and XL2DataIn 1241 follow a WAM load network path to reach theXL1DataOut 1215 and XLO2DataOut 1216, respectively.

By placing the load and store execute units 1234 and 1224 in closeproximity to the memory 1226, the load address lines 1235 and 1236 andstore address lines 1225 do not have to pass through the network 1206.The control signals XL1 cntls 1237, XL2 cntls 1238, and XScntls 1228 areused for multiplexer control in network 1206.

FIG. 12B illustrates processor node 1250 made up of an execution node1252 and a memory node 1254 in accordance with an embodiment of thepresent invention. The execution node 1252 does not have a decoder andreceives decoded arithmetic/logic instruction control signals 1256 froman external instruction decoder such as decoder 1260. The memory node1254 does not have a decoder and receives decoded store and loadinstructions control signals 1257 and 1258, respectively, from anexternal instruction decoder such as decoder 1260. The store and loadinstruction control signals 1257 and 1258 are received in port latch andcontrol units 1262 and 1264, respectively. The port latch and controlunit 1262 supplies the XScntls 1266 to a network 1270. The port latchand control unit 1264 supplies the XL1 cntls 1268 and XL2 cntls 1269 tothe network 1270. The port latch and control unit 1262 supplies thewrite address 1272 to a multiport memory, such as memory 1276. Datareceived from the network on XSDataOut 1282 is stored in the multiportmemory 1276. The port latch and control unit 1264 supplies the readaddresses 1273 and 1274 to a multiport memory, such as the memory 1276to access two data values. The data values are supplied to the networkon XL1DataIn 1283 and XL2DataIn 1284. In this fashion, singleinstructions, such as instructions 1285 may be separately decoded anduse the features and advantages of the present invention.

While the present invention is disclosed in a presently preferredcontext, it will be recognized that the teachings of the presentinvention may be variously embodied consistent with the disclosure andclaims. By way of example, the present invention is applicable toregister based RISC type processors acting as the processor nodes thatcommunicate through a shared global memory. In another example, thenetwork 1206 of FIG. 12A may be implemented with various types ofnetworks while maintaining the split organization of the processor node1200 embodiment of the present invention. It will be recognized that thepresent teachings may be adapted to other present and futurearchitectures to which they may be beneficial.

1. A network of nodes identified according to a G×H 2-dimension matrixof nodes having nearest neighbor connectivity between adjacent nodes ineach dimension which includes wrap around adjacent nodes, the networkcomprising: groups of A_(g,h) nodes, each group having a different gthat is the same for each A_(g,h) node in that group, gε{0, 1, . . . ,G−1} and for each group, hε{0, 1, . . . , H−1}, and each A_(g,h) nodeoperable to output an A_(g,h) data value; groups of R_(g,h) nodes, eachgroup having a different g that is the same for each R_(g,h) node inthat group, gε{0, 1, . . . , G−1} and for each group, hε{0, 1, . . . ,H−1}, each group of R_(g,h) nodes coupled to a corresponding group ofA_(g,h) nodes according to an adjacency of nodes in a first dimension,wherein each R_(g,h) node is operable to select an A_(g,h) data valuefrom a coupled A_(g,h) node and to output the selected A_(g,h) datavalue as an R_(g,h) data value; and groups of S_(g,h) nodes, each grouphaving a different g that is the same for each S_(g,h) node in thatgroup, gε{0, 1, . . . , G−1} and for each group, hε{0, 1, . . . , H−1},each group of S_(g,h) nodes coupled to three groups of R_(g,h) nodesaccording to an adjacency of nodes in a second dimension, wherein eachS_(g,h) node is operable to select an R_(g,h) data value from a coupledR_(g,h) node and to output the selected R_(g,h) data value as an S_(g,h)data value.
 2. The network of claim 1, further comprises: a plurality ofB_(g,h) nodes, gε{0, 1, . . . , G−1} and hε{0, 1, . . . , H−1}, eachB_(g,h) node coupled to a corresponding S_(g,h) node.
 3. The network ofclaim 2, wherein each B_(g,h) node is overlaid upon a correspondingA_(g,h) node.
 4. The network of claim 1, wherein each A_(g,h) node is aprocessor that is operable to select a path through a coupledcorresponding R_(g,h) node to communicate the R_(g,h) data value and toselect a path through a coupled corresponding S_(g,h) node tocommunicate the S_(g,h) data value in response to the A_(g,h) processorexecuting a memory access instruction.
 5. The network of claim 1,wherein each A_(g,h) node is a storage node that is operable to select apath through a coupled corresponding R_(g,h) node to communicate theR_(g,h) data value and to select a path through a coupled correspondingS_(g,h) node to communicate the S_(g,h) data value in response to theA_(g,h) storage node executing a memory access instruction.
 6. Thenetwork of claim 1, wherein the R_(g,h) nodes and the S_(g,h) nodescomprise: a function unit operable to execute a function on data valuesreceived from the coupled nodes.
 7. The network of claim 2, wherein eachB_(g,h) node is a processor that is operable to select a path through acoupled corresponding R_(g,h) nodes to communicate the R_(g,h) datavalue and to select a path through a coupled corresponding S_(g,h) nodesto communicate the S_(g,h) data value in response to the B_(g,h)processor executing a memory access instruction.
 8. The network of claim2, wherein each B_(g,h) node is a storage node that is operable toselect a path through a coupled corresponding R_(g,h) nodes tocommunicate the R_(g,h) data value and to select a path through acoupled corresponding S_(g,h) nodes to communicate the S_(g,h) datavalue in response to the B_(g,h) storage node executing a memory accessinstruction.
 9. The network of claim 1, wherein each R_(g,h) nodecomprises: a multiplexer that is operable to select an A_(g,h) datavalue from a coupled A_(g,h) node and to output the selected A_(g,h)data value as an R_(g,h) data value.
 10. The network of claim 1, whereineach R_(g,h) node comprises a first multiplexer operable to select afirst A_(g,h) data value from a first coupled A_(g,h) node and to outputthe selected first A_(g,h) data value as a first R_(g,h) data value, asecond multiplexer operable to select a second A_(g,h) data value from asecond coupled A_(g,h) node and to output the selected second A_(g,h)data value as a second R_(g,h) data value, and a third multiplexeroperable to select a third A_(g,h) data value from a third coupledA_(g,h) node and to output the selected third A_(g,h) data value as athird R_(g,h) data value.
 11. The network of claim 1 further comprises:an expanded network of nodes identified according to a G×H×K 3-dimensionmatrix, wherein the groups of A_(g,h) nodes, the groups of R_(g,h)nodes, and the groups of S_(g,h) nodes are replicated and identified asgroups of A_(g,h,k) nodes, groups of R_(g,h,k) nodes, and groups ofS_(g,h,k) nodes having a different k that is the same for each node inthat group, kε{0, 1, . . . , K−1} and for each group gε{0, 1, . . . ,G−1} and hε{0, 1, . . . , H−1}; and groups of V_(g,h,k) nodes having adifferent k that is the same for each V_(g,h,k) node in that group,kε{0, 1, . . . , K−1} and for each group gε{0, 1, . . . , G−1} and hε{0,1, . . . , H−1}, each group of V_(g,h,k) nodes coupled to three groupsof S_(g,h,k) nodes according to an adjacency of nodes in a thirddimension, wherein each V_(g,h,k) node is operable to select anS_(g,h,k) data value from a coupled S_(g,h,k) node and to output theselected S_(g,h,k) data value as an V_(g,h,k) data value.
 12. A methodof constructing a 3-dimensional network of nodes N(i)_(g,h,k) hε{0, 1, .. . , H−1}, gε{0, 1, . . . , G−1}, and kε{0, 1, . . . , K−1}, the methodcomprising: coupling for i=0 an output of each node N(i)_(g,h,k) to aninput of node N(i+1)_(g,h,k) and to an input of node N(i+1)_(g,h+1,k)and to an input of node N(i+1)_(g,h−1,k) where h+1 wraps to 0 when h+1=Hand h−1 wraps to H−1 when h−1=−1; coupling for i=1 an output of eachnode N(i)_(g,h,k) to an input of node N(i+1)_(g,h,k) and to an input ofnode N(i+1)_(g+1,h,k) and to an input of node N(i+1)_(g−1,h,k) where g+1wraps to 0 when g+1=G and g−1 wraps to G−1 when g−1=−1; and coupling fori=2 an output of each node N(i)_(g,h,k) to an input of nodeN(i+1)_(g,h,k) and to an input of node N(i+1)_(g,h,k+1) and to an inputof node N(i+1)_(g,h,k−1) where k+1 wraps to 0 when k+1=K and k−1 wrapsto K−1 when k−1=−1, wherein the N(0)_(g,h,k) nodes are operable tocommunicate data values according to a selected network path to anoutput of the N(3)_(g,h,k) nodes.
 13. The method of claim 12 furthercomprises: coupling for i=3 an output of each node N(i)_(g,h,k) to aninput of a node N(i+1)_(g,h,k) wherein N(4)_(g,h,k) identifies a fifthset of nodes as memory nodes and the memory nodes determine the selectednetwork path in response to each memory node executing a memory accessinstruction.
 14. The method of claim 12 further comprises: coupling nodecontrol signals from the N(0)_(g,h,k) nodes in the first set of nodes,to the N(1)_(g,h,k) nodes in the second set of nodes, to theN(2)_(g,h,k) nodes in the third set of nodes, and to the N(3)_(g,h,k)nodes in the fourth set of nodes; and determining the selected networkpath based on the node control signals.
 15. A network of nodes organizedaccording to a multi-dimension matrix of nodes having nearest neighborconnectivity between adjacent nodes in each dimension, the networkcomprising: a plurality of A nodes, each A node identified according toits position in the multi-dimension matrix of nodes and operable tooutput an A data value; a plurality of R nodes, each A node coupled tothree R nodes that are adjacent to the A node in a first dimension ofthe multi-dimension matrix of nodes, wherein each coupled R node isoperable to select an A data value from a coupled A node and to outputthe selected A value as an R data value; and a plurality of S nodes,each R node coupled to three S nodes that are adjacent to the R node ina second dimension of the multi-dimension matrix of nodes, wherein eachcoupled S node is operable to select an R data value from a coupled Rnode and to output the selected R data value as an S data value.
 16. Thenetwork of claim 15, further comprises: a plurality of V nodes, each Snode coupled to three V nodes that are adjacent to the S node in a thirddimension of the multi-dimension matrix of nodes, wherein each coupled Vnode is operable to provide complex operations on multiple coupled Sdata values.
 17. The network of claim 16, wherein each A node is coupledto a set of nine S nodes and to a set of twenty seven V nodes, and eachadditional dimension of nodes expands the set of nodes each A node iscoupled to by a multiple of three.
 18. A network of nodes identifiedaccording to a G×H 2-dimension matrix of nodes, the network comprising:a plurality of A_(g,h) nodes gε{0, 1, . . . , G−1} and hε{0, 1, . . . ,H−1}, each A_(g,h) node operable to output an A_(g,h) data value; aplurality of R_(g,h) nodes gε{0, 1, . . . , G−1} and hε{0, 1, . . . ,H−1}, and each A_(g,h) node is coupled to a corresponding R_(g,h) node,to an R_(g,h−1) node, and to an R_(g,h+1) node, where h+1 wraps to 0when h+1=H and h−1 wraps to H−1 when h−1=−1, wherein each coupledR_(g,h) node is operable to select an A_(g,h) data value from a coupledA_(g,h) node and to output the selected A_(g,h) data value as an R_(g,h)data value; and a plurality of S_(g,h) nodes gε{0, 1, . . . , G−1} andhε{0, 1, . . . , H−1}, the corresponding R_(g,h) node is coupled to acorresponding S_(g,h) node, to an S_(g−1,h) node, and to an S_(g+1,h)node, the R_(g,h−1) node is coupled to a corresponding S_(g,h−1) node,to an S_(g−1,h−1) node, and to an S_(g+1,h−1) node, and the R_(g,h+1)node is coupled to a corresponding S_(g,h+1) node, to an S_(g−1,h+1)node, and to an S_(g+1,h+1) node, where h+1 wraps to 0 when h+1=H andh−1 wraps to H−1 when h−1=−1, wherein each coupled S_(g,h) node isoperable to select an R_(g,h) data value from a coupled R_(g,h) node andto output the selected R_(g,h) data value as an S_(g,h) data value. 19.The network of claim 18, wherein each R_(g,h) node comprises a firstmultiplexer operable to select a first A_(g,h) data value from a firstcoupled A_(g,h) node and to output on a first output port the selectedfirst A_(g,h) data value as a first R_(g,h) data value, a secondmultiplexer operable to select a second A_(g,h) data value from a secondcoupled A_(g,h) node and to output on a second output port the selectedsecond A_(g,h) data value as a second R_(g,h) data value, and a thirdmultiplexer operable to select a third A_(g,h) data value from a thirdcoupled A_(g,h) node and to output on a third output port the selectedthird A_(g,h) data value as a third R_(g,h) data value.
 20. The networkof claim 19, wherein for the corresponding R_(g,h) node, the firstoutput port is coupled to the corresponding S_(g,h) node, the secondoutput port is coupled to the S_(g−1,h) node, and the third output portis couple to the S_(g+1,h) node.